Papers with intermediate task
When to Use Multi-Task Learning vs Intermediate Fine-Tuning for Pre-Trained Encoder Transfer Learning (2022.acl-short)
Copied to clipboard
| Challenge: | Transfer learning (TL) in natural language processing has seen a surge of interest in recent years . pre-trained models have shown impressive ability to transfer to novel tasks . |
| Approach: | They compare two different methods of transfer learning in natural language processing to find out which is better. |
| Outcome: | The proposed methods perform better when the target task has fewer instances than the supporting task and vice versa. |
English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Too (2020.aacl-main)
Copied to clipboard
Jason Phang, Iacer Calixto, Phu Mon Htut, Yada Pruksachatkun, Haokun Liu, Clara Vania, Katharina Kann, Samuel R. Bowman
| Challenge: | a study of intermediate-task training in monolingual English shows that it improves model performance on non-English language understanding tasks. |
| Approach: | They evaluate whether English intermediate-task training is still helpful on non-English target tasks . BUCC and Tatoeba sentence retrieval tasks see large improvements . |
| Outcome: | The proposed model outperforms existing models on non-English language understanding tasks. |
Rethinking Why Intermediate-Task Fine-Tuning Works (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained language models can be fine-tuned on intermediate labeled-data tasks before fine- tuning the models on the target task of interest. |
| Approach: | They conduct extensive experiments to study the impact of different factors on STILT . they find that the improvement from an intermediate task could be orthogonal to it containing reasoning or other complex skills. |
| Outcome: | The proposed method improves the performance of pretrained language models on various target tasks. |
Divergence-Based Domain Transferability for Zero-Shot Classification (2023.findings-eacl)
Copied to clipboard
| Challenge: | a recent study shows that fine-tuning of neural models can improve performance on language-based tasks without brute-force searching effective task combinations. |
| Approach: | They propose to use divergence measures to estimate whether one task pair will perform better than another . they use 58 tasks and 6,600 task pair combinations to study the effect of different tuning methods . |
| Outcome: | The proposed method reduces end-to-end runtime by 40% by estimating transferability . the proposed method is based on 58 tasks and over 6,600 task pair combinations . |
Beyond Distributional Hypothesis: Let Language Models Learn Meaning-Text Correspondence (2022.findings-naacl)
Copied to clipboard
| Challenge: | Recent evidence shows that large-size pre-trained language models do not satisfy the logical negation property (LNP) However, their reliability is being challenged due to faulty behaviours and incomprehension on number-related representations. |
| Approach: | They propose a new intermediate training task to directly learn meaning text correspondence instead of relying on the distributional hypothesis. |
| Outcome: | The proposed approach outperforms previous models on 7 GLUE tasks and outperformed previous models. |
A Tale of Two Revisions: Summarizing Changes Across Document Versions (2024.findings-acl)
Copied to clipboard
| Challenge: | Document revision is a crucial aspect of the writing process, especially in collaborative environments where multiple authors contribute simultaneously. |
| Approach: | They propose a task of providing thematic summary of changes between document versions, organizing individual edits based on shared themes, and propose three strategies to tackle this task. |
| Outcome: | The proposed model improves its capacity to handle the task and also enables it to be used in a curated dataset. |
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (P19-1)
Copied to clipboard
Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, Shuning Jin, Berlin Chen, Benjamin Van Durme, Edouard Grave, Ellie Pavlick, Samuel R. Bowman
| Challenge: | State-of-the-art models in natural language processing (NLP) often incorporate sentence encoder functions which generate a sequence of vectors intended to represent the in-context meaning of each word in an input text. |
| Approach: | They conduct the first large-scale systematic study of candidate pretraining tasks, comparing 19 different tasks as alternatives and complements to language modeling. |
| Outcome: | The proposed model can be used to train sentences on language modeling tasks. |
Cluster & Tune: Boost Cold Start Performance in Text Classification (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods to fine-tune pre-trained models for text classification are poor in practice. |
| Approach: | They propose to add an intermediate unsupervised classification task between pre-training and fine-tuning phases to boost performance of pre-trained models. |
| Outcome: | The proposed method improves performance on topical classification tasks when labeled data is scarce. |
Why Is It Hate Speech? Masked Rationale Prediction for Explainable Hate Speech Detection (2022.coling-1)
Copied to clipboard
| Challenge: | Hate speech cannot be identified based solely on the presence of specific words; model should reason like humans and be explainable. |
| Approach: | They propose to use Masked Rationale Prediction to predict masked human rationales . the method performs hate speech detection robustly in terms of bias and explainability . |
| Outcome: | The proposed method performs state-of-the-art in terms of bias and explainability. |
French Tweet Corpus for Automatic Stance Detection (2020.lrec-1)
Copied to clipboard
| Challenge: | a new corpus of tweets is being developed for automatic stance detection of fake news . the task involves determining the attitude expressed in a text toward a target . this is a difficult task to overcome as discussions about fake news are controversial . |
| Approach: | They propose to build a human-annotated corpus for automatic stance detection of tweets in french . they propose to use four classes broadly adopted by the community for annotation . |
| Outcome: | The proposed corpus is the first freely available stance annotated tweet corpus in the french language. |
Linguistic Knowledge Can Enhance Encoder-Decoder Models (If You Let It) (2024.lrec-main)
Copied to clipboard
| Challenge: | a recent study has shown that pre-trained NLMs can capture syntax- and semantic-sensitive phenomena. |
| Approach: | They investigate whether fine-tuning pre-trained models with linguistic knowledge improves their performance in a target task. |
| Outcome: | The proposed enhancements improve models' performance in a target task, the authors show . the study includes models in Italian and English, and multilingual models in English and Italian . |
On an Intermediate Task for Classifying URL Citations on Scholarly Papers (2024.lrec-main)
Copied to clipboard
| Challenge: | Citations using URLs can be used as information source for research resource search engines. |
| Approach: | They propose a method to classify URL citations using a simple fine-tuning strategy. |
| Outcome: | The proposed method outperforms methods using a simple fine-tuning strategy with higher macro F-scores for different model sizes and architectures. |